Potential of Large Language Models in Health Care: Delphi Study.

BACKGROUND: A large language model (LLM) is a machine learning model inferred from text data that captures subtle patterns of language use in context. Modern LLMs are based on neural network architectures that incorporate transformer methods. They allow the model to relate words together through attention to multiple words in a text sequence. LLMs have been shown to be highly effective for a range of tasks in natural language processing (NLP), including classification and information extraction tasks and generative applications. OBJECTIVE: The aim of this adapted Delphi study was to collect researchers' opinions on how LLMs might influence health care and on the strengths, weaknesses, opportunities, and threats of LLM use in health care. METHODS: We invited researchers in the fields of health informatics, nursing informatics, and medical NLP to share their opinions on LLM use in health care. We started the first round with open questions based on our strengths, weaknesses, opportunities, and threats framework. In the second and third round, the participants scored these items. RESULTS: The first, second, and third rounds had 28, 23, and 21 participants, respectively. Almost all participants (26/28, 93% in round 1 and 20/21, 95% in round 3) were affiliated with academic institutions. Agreement was reached on 103 items related to use cases, benefits, risks, reliability, adoption aspects, and the future of LLMs in health care. Participants offered several use cases, including supporting clinical tasks, documentation tasks, and medical research and education, and agreed that LLM-based systems will act as health assistants for patient education. The agreed-upon benefits included increased efficiency in data handling and extraction, improved automation of processes, improved quality of health care services and overall health outcomes, provision of personalized care, accelerated diagnosis and treatment processes, and improved interaction between patients and health care professionals. In total, 5 risks to health care in general were identified: cybersecurity breaches, the potential for patient misinformation, ethical concerns, the likelihood of biased decision-making, and the risk associated with inaccurate communication. Overconfidence in LLM-based systems was recognized as a risk to the medical profession. The 6 agreed-upon privacy risks included the use of unregulated cloud services that compromise data security, exposure of sensitive patient data, breaches of confidentiality, fraudulent use of information, vulnerabilities in data storage and communication, and inappropriate access or use of patient data. CONCLUSIONS: Future research related to LLMs should not only focus on testing their possibilities for NLP-related tasks but also consider the workflows the models could contribute to and the requirements regarding quality, integration, and regulations needed for successful implementation in practice.

Large Language Model-Based Evaluation of Medical Question Answering Systems: Algorithm Development and Case Study.

BACKGROUND: Healthcare systems are increasingly resource constrained, leaving less time for important patient-provider interactions. Conversational agents (CAs) could be used to support the provision of information and to answer patients' questions. However, information must be accessible to a variety of patient populations, which requires understanding questions expressed at different language levels. METHODS: This study describes the use of Large Language Models (LLMs) to evaluate predefined medical content in CAs across patient populations. These simulated populations are characterized by a range of health literacy. The evaluation framework includes both fully automated and semi-automated procedures to assess the performance of a CA. RESULTS: A case study in the domain of mammography shows that LLMs can simulate questions from different patient populations. However, the accuracy of the answers provided varies depending on the level of health literacy. CONCLUSIONS: Our scalable evaluation framework enables the simulation of patient populations with different health literacy levels and helps to evaluate domain specific CAs, thus promoting their integration into clinical practice. Future research aims to extend the framework to CAs without predefined content and to apply LLMs to adapt medical information to the specific (health) literacy level of the user.

Analysis of Critical Incident Reports Using Natural Language Processing.

A Critical Incident Reporting System (CIRS) collects anecdotal reports from employees, which serve as a vital source of information about incidents that could potentially harm patients. OBJECTIVES: To demonstrate how natural language processing (NLP) methods can help in retrieving valuable information from such incident data. METHODS: We analyzed frequently occurring terms and sentiments as well as topics in data from the Swiss National CIRRNET database from 2006 to 2023 using NLP and BERTopic modelling. RESULTS: We grouped the topics into 10 major themes out of which 6 are related to medication. Overall, they reflect the global trends in adverse events in healthcare (surgical errors, venous thromboembolism, falls). Additionally, we identified errors related to blood testing, COVID-19, handling patients with diabetes and pediatrics. 40-50% of the messages are written in a neutral tone, 30-40% in a negative tone. CONCLUSION: The analysis of CIRS messages using text analysis tools helped in getting insights into common sources of critical incidents in Swiss healthcare institutions. In future work, we want to study more closely the relations, for example between sentiment and topics.

Transformer Models in Healthcare: A Survey and Thematic Analysis of Potentials, Shortcomings and Risks.

Large Language Models (LLMs) such as General Pretrained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT), which use transformer model architectures, have significantly advanced artificial intelligence and natural language processing. Recognized for their ability to capture associative relationships between words based on shared context, these models are poised to transform healthcare by improving diagnostic accuracy, tailoring treatment plans, and predicting patient outcomes. However, there are multiple risks and potentially unintended consequences associated with their use in healthcare applications. This study, conducted with 28 participants using a qualitative approach, explores the benefits, shortcomings, and risks of using transformer models in healthcare. It analyses responses to seven open-ended questions using a simplified thematic analysis. Our research reveals seven benefits, including improved operational efficiency, optimized processes and refined clinical documentation. Despite these benefits, there are significant concerns about the introduction of bias, auditability issues and privacy risks. Challenges include the need for specialized expertise, the emergence of ethical dilemmas and the potential reduction in the human element of patient care. For the medical profession, risks include the impact on employment, changes in the patient-doctor dynamic, and the need for extensive training in both system operation and data interpretation.

Digital Interventions and Their Unexpected Outcomes - Time for Digitalovigilance?

The application of digital interventions in healthcare beyond research has been translated in the development of software as a medical device. Along with corresponding regulations for medical devices, there is a need for assessing adverse events to conduct post-market surveillance and to appropriately label digital health interventions to ensure proper use and patient safety. To date unexpected consequences of digital health interventions are neglected or ignored, or at least remain undescribed in literature. This paper is intended to raise awareness across the research community about these upcoming challenges. We recommend that - together with developing a new research field of digitalovigilance - a systematic assessment and monitoring of adverse events and unexpected interactions be included in clinical trials, along with the reporting of such events and the conduct of meta-analyses on critical aspects.

Large language model-based information extraction from free-text radiology reports: a scoping review protocol.

INTRODUCTION: Radiological imaging is one of the most frequently performed diagnostic tests worldwide. The free-text contained in radiology reports is currently only rarely used for secondary use purposes, including research and predictive analysis. However, this data might be made available by means of information extraction (IE), based on natural language processing (NLP). Recently, a new approach to NLP, large language models (LLMs), has gained momentum and continues to improve performance of IE-related tasks. The objective of this scoping review is to show the state of research regarding IE from free-text radiology reports based on LLMs, to investigate applied methods and to guide future research by showing open challenges and limitations of current approaches. To our knowledge, no systematic or scoping review of IE from radiology reports based on LLMs has been published. Existing publications are outdated and do not comprise LLM-based methods. METHODS AND ANALYSIS: This protocol is designed based on the JBI Manual for Evidence Synthesis, chapter 11.2: 'Development of a scoping review protocol'. Inclusion criteria and a search strategy comprising four databases (PubMed, IEEE Xplore, Web of Science Core Collection and ACM Digital Library) are defined. Furthermore, we describe the screening process, data charting, analysis and presentation of extracted data. ETHICS AND DISSEMINATION: This protocol describes the methodology of a scoping literature review and does not comprise research on or with humans, animals or their data. Therefore, no ethical approval is required. After the publication of this protocol and the conduct of the review, its results are going to be published in an open access journal dedicated to biomedical informatics/digital health.

How Participatory Health Informatics Catalyzes One Digital Health.

OBJECTIVE: To identify links between Participatory Health Informatics (PHI) and the One Digital Health framework (ODH) and to show how PHI could be used as a catalyst or contributor to ODH. METHODS: We have analyzed the addressed topics within the ODH framework in previous IMIA Yearbook contributions from our working group during the last 10 years. We have matched main themes with the ODH's framework three perspectives (individual health and wellbeing, population and society, and ecosystem). RESULTS: PHI catalysts ODH individual health and wellbeing perspective by providing a more comprehensive view on human health, attitudes, and relations between human health and animal health. Integration of specific behavior change techniques or gamification strategies in digital solutions are effective to change behaviors which address the P5 paradigm. PHI supports the population and society perspective through the engagement of the various stakeholders in healthcare. At the same time, PHI might increase a risk for health inequities due to technologies inaccessible to all equally and challenges associated with this. PHI is a catalyst for the ecosystem perspective by contributing data into the digital health data ecosystem allowing for analysis of interrelations between the various data which in turn might provide links among all components of the healthcare ecosystem. CONCLUSION: Our results suggest that PHI can and will involve topics relating to ODH. As the ODH concept crystalizes and becomes increasingly influential, its themes will permeate and become embedded in PHI even more. We look forward to these developments and co-evolution of the two frameworks.

Exploring the Evolution of Social Media in Mental Health Interventions: A Mapping Review.

BACKGROUND: With the rise of social media, social media use for delivering mental health interventions has become increasingly popular. However, there is no comprehensive overview available on how this field developed over time. OBJECTIVES: The objective of this paper is to provide an overview over time of the use of social media for delivering mental health interventions. Specifically, we examine which mental health conditions and target groups have been targeted, and which social media channels or tools have been used since this topic first appeared in research. METHODS: To provide an overview of the use of social media for mental health interventions, we conducted a search for studies in four databases (PubMed; ACM Digital Library; PsycInfo; and CINAHL) and two trial registries (Clinicaltrials.gov; and Cochranelibrary.com). A sample of representative keywords related to mental health and social media was used for that search. Automatic text analysis methods (e.g., BERTopic analysis, word clouds) were applied to identify topics, and to extract target groups and types of social media. RESULTS: A total of 458 studies were included in this review (n=228 articles, and n=230 registries). Anxiety and depression were the most frequently mentioned conditions in titles of both articles and registries. BERTopic analysis identified depression and anxiety as the main topics, as well as several addictions (including gambling, alcohol, and smoking). Mental health and women's research were highlighted as the main targeted topics of these studies. The most frequently targeted groups were "adults" (39.5%) and "parents" (33.4%). Facebook, WhatsApp, messenger platforms in general, Instagram, and forums were the most frequently mentioned tools in these interventions. CONCLUSIONS: We learned that research interest in social media-based interventions in mental health is increasing, particularly in the last two years. A variety of tools have been studied, and trends towards forums and Facebook show that tools allowing for more content are preferred for mental health interventions. Future research should assess which social media tools are best suited in terms of clinical outcomes. Additionally, we conclude that natural language processing tools can help in studying trends in research on a particular topic.

How Can Transformer Models Shape Future Healthcare: A Qualitative Study.

Transformer models have been successfully applied to various natural language processing and machine translation tasks in recent years, e.g. automatic language understanding. With the advent of more efficient and reliable models (e.g. GPT-3), there is a growing potential for automating time-consuming tasks that could be of particular benefit in healthcare to improve clinical outcomes. This paper aims at summarizing potential use cases of transformer models for future healthcare applications. Precisely, we conducted a survey asking experts on their ideas and reflections for future use cases. We received 28 responses, analyzed using an adapted thematic analysis. Overall, 8 use case categories were identified including documentation and clinical coding, workflow and healthcare services, decision support, knowledge management, interaction support, patient education, health management, and public health monitoring. Future research should consider developing and testing the application of transformer models for such use cases.

Personalized Digital Solutions for Mental Health.

INTRODUCTION: Mental health is one of the major global concerns in the field of healthcare. The emergence of digital solutions is proving to be a great aid for individuals suffering from mental health disorders. These solutions are particularly useful and effective when they are personalized. The objective of this paper is to understand the personalization factors and the methods that have been used to collect information to personalize the digital mental health solutions. METHODS: This paper builds on a previous review that analyzed the personalization of digital solutions in mHealth, and expands on the extracted information for the specific case of mental health. RESULTS: Ten mental health digital solutions have been analyzed. The paper focuses on targeted conditions, personalization factors and the methods used for collecting personalization factors. DISCUSSION: The analyzed mental health digital solutions cover a wide range of health conditions. It is remarkable that most articles do not explicitly mention the factors used to personalize the solution. Among the solutions that mention them, there is a great diversity of factors utilized, such as age, gender, user preferences, and subjective behavior. The authors point out the methods for obtaining data to personalize the solutions, including in-app questionnaires, self-reports, and usage data of the solutions. CONCLUSIONS: The analysis of current mental health digital solutions emphasizes the need to create guidelines for designing personalized digital solutions for mental health.

Assessing the Potential Risks of Digital Therapeutics (DTX): The DTX Risk Assessment Canvas.

MOTIVATION: Digital therapeutics (DTX), i.e., health interventions that are provided through digital means, are increasingly available for use; in some countries, physicians can even prescribe selected DTX following a reimbursement by health insurances. This results in an increasing need for methodologies to consider and monitor DTX's negative consequences, their risks to patient safety, and possible adverse events. However, it is completely unknown which aspects should be subject to surveillance given the missing experiences with the tools and their negative impacts. OBJECTIVE: Our aim is to develop a tool-the DTX Risk Assessment Canvas-that enables researchers, developers, and practitioners to reflect on the negative consequences of DTX in a participatory process. METHOD: Taking the well-established business model canvas as a starting point, we identified relevant aspects to be considered in a risk assessment of a DTX. The aspects or building blocks of the canvas were constructed in a two-way process: first, we defined the aspects relevant for discussing and reflecting on how a DTX might bring negative consequences and risks for its users by considering ISO/TS 82304-2, the scientific literature, and by reviewing existing DTX and their listed adverse effects. The resulting aspects were grouped into thematic blocks and the canvas was created. Second, six experts in health informatics and mental health provided feedback and tested the understandability of the initial canvas by individually applying it to a DTX of their choice. Based on their feedback, the canvas was modified. RESULTS: The DTX Risk Assessment Canvas is organized into 15 thematic blocks which are in turn grouped into three thematic groups considering the DTX itself, the users of the DTX, and the effects of the DTX. For each thematic block, questions have been formulated to guide the user of the canvas in reflecting on the single aspects. Conclusions: The DTX Risk Assessment Canvas is a tool to reflect the negative consequences and risks of a DTX by discussing different thematic blocks that together constitute a comprehensive interpretation of a DTX regarding possible risks. Applied during the DTX design and development phase, it can help in implementing countermeasures for mitigation or means for their monitoring.

Coadaptation Between Smart Technologies and Older Adults Over Time: Protocol for a Scoping Review.

BACKGROUND: The Internet of Things (IoT) has gained significant attention due to advancements in technology and has potential applications in meeting the needs of an aging population. Smart technologies, a subset of IoT, can support older adults in aging in place, promoting independent living and improving their quality of life. However, there is a lack of research on how older adults and smart technologies coadapt over time to maximize their benefits and sustain adoption. OBJECTIVE: We will aim to comprehensively review and analyze the existing scientific literature pertaining to the coadaptation between smart technologies and older adults. The primary focus will be to investigate the extent and nature of this coadaptation process and explore how older adults and technology coevolve over time to enhance older adults' experience with technology. METHODS: This scoping review will follow the methodology outlined in the Joanna Briggs Institute Reviewer's Manual and adhere to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analysis Extension for Scoping Reviews) guidelines for reporting. Peer-reviewed articles will be searched in databases like Ovid MEDLINE, OVID Embase, PEDro, OVID PsycINFO, EBSCO CINAHL, the Cochrane Library, Scopus, IEEE Xplore, Web of Science, and Global Index Medicus. The research team will create a data extraction form covering study characteristics, participant characteristics, underlying models and frameworks, research findings, implications for technology coadaptation, and any identified study limitations. A directed content analysis approach will be used, incorporating the Selection, Optimization, and Compensation framework and Sex- and Gender-Based Analysis Plus theoretical framework. RESULTS: The results of this study are expected in January 2024. CONCLUSIONS: This scoping review endeavors to present a thorough overview of the available evidence concerning how smart technologies interact with older adults over an extended period. The insights gained from this review will lay the groundwork for a research program that explores how older adults adapt to and use smart technologies throughout their lives, ultimately leading to improved user satisfaction and experience and facilitating aging in place with tailored support and user-centered design principles. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/51129.

Designing personalised mHealth solutions: An overview.

INTRODUCTION: Mobile health, or mHealth, is based on mobile information and communication technologies and provides solutions for empowering individuals to participate in healthcare. Personalisation techniques have been used to increase user engagement and adherence to interventions delivered as mHealth solutions. This study aims to explore the current state of personalisation in mHealth, including its current trends and implementation. MATERIALS AND METHODS: We conducted a review following PRISMA guidelines. Four databases (PubMed, ACM Digital Library, IEEE Xplore, and APA PsycInfo) were searched for studies on mHealth solutions that integrate personalisation. The retrieved papers were assessed for eligibility and useful information regarding integrated personalisation techniques. RESULTS: Out of the 1,139 retrieved studies, 62 were included in the narrative synthesis. Research interest in the personalisation of mHealth solutions has increased since 2020. mHealth solutions were mainly applied to endocrine, nutritional, and metabolic diseases; mental, behavioural, or neurodevelopmental diseases; or the promotion of healthy lifestyle behaviours. Its main purposes are to support disease self-management and promote healthy lifestyle behaviours. Mobile applications are the most prevalent technological solution. Although several design models, such as user-centred and patient-centred designs, were used, no specific frameworks or models for personalisation were followed. These solutions rely on behaviour change theories, use gamification or motivational messages, and personalise the content rather than functionality. A broad range of data is used for personalisation purposes. There is a lack of studies assessing the efficacy of these solutions; therefore, further evidence is needed. DISCUSSION: Personalisation in mHealth has not been well researched. Although several techniques have been integrated, the effects of using a combination of personalisation techniques remain unclear. Although personalisation is considered a persuasive strategy, many mHealth solutions do not employ it. CONCLUSIONS: Open research questions concern guidelines for successful personalisation techniques in mHealth, design frameworks, and comprehensive studies on the effects and interactions among multiple personalisation techniques.

The performance of serious games for enhancing attention in cognitively impaired older adults.

Attention, which is the process of noticing the surrounding environment and processing information, is one of the cognitive functions that deteriorate gradually as people grow older. Games that are used for other than entertainment, such as improving attention, are often referred to as serious games. This study examined the effectiveness of serious games on attention among elderly individuals suffering from cognitive impairment. A systematic review and meta-analyses of randomized controlled trials were carried out. A total of 10 trials ultimately met all eligibility criteria of the 559 records retrieved. The synthesis of very low-quality evidence from three trials, as analyzed in a meta-study, indicated that serious games outperform no/passive interventions in enhancing attention in cognitively impaired older adults (P < 0.001). Additionally, findings from two other studies demonstrated that serious games are more effective than traditional cognitive training in boosting attention among cognitively impaired older adults. One study also concluded that serious games are better than traditional exercises in enhancing attention. Serious games can enhance attention in cognitively impaired older adults. However, given the low quality of the evidence, the limited number of participants in most studies, the absence of some comparative studies, and the dearth of studies included in the meta-analyses, the results remain inconclusive. Thus, until the aforementioned limitations are rectified in future research, serious games should serve as a supplement, rather than a replacement, to current interventions.

How Does Shortage of Health IT Professionals Impact on the Digital Health Transformation?

BACKGROUND: The need for software suppliers to react swiftly to the plethora of application requests and constantly shifting market requirements is one of the major problems facing the health IT business in the context of digital health transformation. This can only be achieved when the necessary staff and resources are available. OBJECTIVES: The objective of this work is to identify challenges health IT companies are confronted with related to personnel capacities and skilled workers. METHODS: Using a questionnaire distributed through newsletters and social media among representatives of software companies and hospitals we collected information on current hurdles of health software providers and their strategies to overcome these in order to address the demands of the digital health transformation. RESULTS: The main findings of the survey are that scarce resources in software development are among the reasons for not achieving strategic goals on time in the health IT sector and for not being able to react flexibly to market changes. A strategy to overcome missing expert knowledge and own resources without free capacity is to hire external resources. CONCLUSIONS: With the ever-changing landscape of digital health, it is essential to have skilled workers with knowledge on the peculiarities of clinical workflows. The existing shortage of skilled workers leads to a reduction of innovative power in the health IT sector, potentially slowing down the digital health transformation.

Designing a Digital Medical Interview Assistant for Radiology.

Radiologists rarely interact with the patients whose radiological images they are reviewing due to time and resource constraints. However, relevant information about the patient's medical history could improve reporting performance and quality. In this work, our objective was to collect requirements for a digital medical interview assistant (DMIA) that collects the medical history from patients by means of a conversational agent and structures as well as provides the collected data to radiologists. Requirements were gathered based on a narrative literature review, a patient questionnaire and input from a radiologist. Based on these results, a system architecture for the DMIA was developed. 37 functional and 17 non-functional requirements were identified. The resulting architecture comprises five components, namely Chatbot, Natural language processing (NLP), Administration, Content Definition and Workflow Engine. To be able to quickly adapt the chatbot content according to the information needs of a specific radiological examination, there is a need for developing a sustainable process for the content generation that considers standardized data modelling as well as rewording of clinical language into consumer health vocabulary understandable to a diverse patient user group.

Fading Fatigue - A Self-Management App for Supporting Long-COVID Patients with Fatigue.

Fatigue is the most prevalent Long-COVID symptom. Individuals who are affected have to learn to organize and manage daily activities according to the subjectively perceived energy reserves. Our objective was to develop an application, Fading Fatigue, that supports patients in their energy management, in particular after an initial therapy guided by health professionals. Fading Fatigue was developed in an iterative approach and implemented as a client-server application. Interviews and a literature search were conducted to identify limitations and challenges of the current treatment. Fading Fatigue offers several tools for energy management: a daily energy planner, a documentation aid for well-being and a progress view. Future work should study usability. Inclusion of additional features increasing the adherence such as providing feedback could be considered.

Towards Safe Conversational Agents in Healthcare.

Conversational agents (CA) are becoming very popular to deliver digital health interventions. These dialog-based systems are interacting with patients using natural language which might lead to misunderstandings and misinterpretations. To avoid patient harm, safety of health CA has to be ensured. This paper raises awareness on safety when developing and distributing health CA. For this purpose, we identify and describe facets of safety and make recommendations for ensuring safety in health CA. We distinguish three facets of safety: 1) system safety, 2) patient safety, and 3) perceived safety. System safety comprises data security and privacy which has to be considered when selecting technologies and developing the health CA. Patient safety is related to risk monitoring and risk management, to adverse events and content accuracy. Perceived safety concerns a user's perception of the level of danger and user's level of comfort during the use. The latter can be supported when data security is guaranteed and relevant information on the system and its capabilities are provided.